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ON-LINE SURVEYS OF USER AND EMPLOYEE SATISFACTION 


1. Abstract 


On-line survey facilities have been developed at the T J Watson Research 
Center (IBM, Yorktown Heights, N.Y.) Their purposes are to evaluate user 
satisfaction with computer services and employee satisfaction with the 
job. These facilities are: 


Surveys at logoff/disconn time about system response time 
On-line opinions surveys of user and employee satisfaction 
A general description of the facilities, implementation, advantages, 


human factors, security considerations, and achieved objectives are out- 
lined in this paper. 
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2. SURVEY CONDUCTED AT LOGOFF/DISCONN TIME 


2.1 Introduction 


The LOGOFFs and DISCONNs commands are intercepted during the last part of 
the working day and the users are asked to rate the system response time. 
Their responses are transmitted in the form of messages to a receiving 
virtual machine. The receiving virtual machine captures these messages 
into a CMS file, and after completion of the daily survey it computes sta- 
tistics and generates reports using the collected data. 


2.2 Objectives 


To obtain from the users an evaluation of prime shift system response time 
for their terminal session. To determine level of service which satisfy 
users and which users need help. To generate statistical reports and 
maintain a history of it. 


2.3 Design Considerations 


The interception of the LOGOFFs and DISCONNs is done by placing CMS execs 
with these names and their valid truncations in the system. This has the 
disadvantage that the LOGOFFs and DISCONNs exercised by the user at the CP 
level are not intercepted. However this route was chosen since it was less 
practical to modify the pertinent system modules. It also offers the users 
the choice of bypassing entirely the survey by simply typing: ‘CP LOGOFF' 
or 'CP DISCONN'. This is desirable specially since the survey is conducted 
daily. 


Naive users 


The survey is carried out in a conversational way and without resorting to 
full screen displays. This avoids situations where users not familiar with 
full screen mode may not know how to proceed. It also eliminates the prob- 
lem of having to take into separate consideration typewriter type termi- 
nals or certain other terminals (like Textronix graphic) that are treated 
as 2741 typewriter terminals by the system. 


Burden on the users 
To avoid burdening the users the LOGOFFs, DISCONNs interception takes 


place at random only in one out of four cases, on business days from 
3:30PM to 6:00PM and is limited to one per user per day. 


a 


Syntax equivalent to CP LOGOFF and CP DISCONN 


Any permissible syntax for the CP logoff and disconn commands can be exer- 
cised when the logoff/disconn execs are invoked. In addition they are 
operational in any environment where the CP commands are valid (Editors, 
Dial PVM etc.) 


Speed of execution 


The design results in a a prompt logoff and disconn when the survey is not 
taken place (Before 3:30PM and after 6:00PM). This is particularly impor- 
tant because any noticeable increase of waiting time when logging off or 
disconnecting would be a frustration unacceptable to the user. 


Security 


A user is disconnected or logged off if he does not answer within a cer- 
tain time the question raised by the execs when interception and surveying 
is taking place. This is done for security reasons to take into account 
the case of somebody logging off (or disconnecting) and walking immediate- 
ly away from the terminal. 


Users running disconnected 


For obvious reasons the survey is bypassed for the case of users running 
disconnected. 


2.4 General description 


There are two main components to the facility, namely: a collection of 
CMS system execs and a receiving virtual machine. 

The first component intercepts the LOGOFFs and DISCONNs commands, asks the 
user to enter a rating of system response time and then proceeds to com- 
plete the command's intended action. 

The second component captures the responses of the users and produces sta- 
tistical reports. 


2.4.1 CMS execs that intercept the LOGOFFs and DISCONNs 


These execs logoff and disconnect a user after asking the following ques- 
tion: "How do you rate today's system response for processing your prime 
shift work?. Type 1 for very good, 2 for good, 3 for fair, 4 for poor, 5 
for very poor, or 0 for no opinion." They do not survey the user if one 
of the following conditions occur: 


1) The day of the week is not a business day 


2) The time of the day is not in the interval (15:30 to 18:00) 


3) The user is running disconnected 

4) The user does not have a R/W minidisk accessed as A 

5) The user has already participated in the survey during that day 
The following additional precaution is taken: 


6) The user is logged off or disconnected if he fails to response 
within 60 seconds 


2.4.2 Receiving virtual machine 


The following are the main functions of this receiving station: 


a) To capture any messages send to it by the users during the time the 
survey is conducted 


b) To compute statistics on the users responses at the conclusion of 
the survey 


c) Maintain on line only information for the last 20 business days 


Daily, on business days, the receiving virtual machine wakes up at 15:25 
and places itself into a receive state during the survey's time (15:25 to 
18:00). It then captures the incoming users responses and places them in 
a CMS file. At 18:00 it leaves the receiving state and generates or modi- 
fies the following files: 


RESPONSE FILE, contains statistics (including the average rating of 
response time) for: the last business day, accumulative month to date, 
for the last 20 business days. This file is shown in figure 1 of Sec- 
tion 2.6 


COMPLAIN FILE, contains a history of the users ratings for the last 20 
business days sorted in descending order from the lowest to the highest 
rating and subsequently by frequency. 


RESPONSE HISTORY, contains a record of the RESPONSE FILE for the last 
20 business days. : 


RESPONSE LOG, contains a record of the users ratings for the last 20 
business days. Extraneous messages, no opinion ratings, and duplicate 
entries deleted. A duplicate entry is a record with the same userid, 
the same rating for the same date. 


After generating these files the virtual machine places itself into a 
sleep state until 15:25 of the next business day. 
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2.5 Graphics obtained by using the collected data 


The average daily user rating of response time was plotted versus some of 
the recorded system measurements. Figures 2 and 3, in Section 2.6, show 
the plots of the average response time rating versus the daily average 
response time and the daily average expansion factor respectively, for the 
VM V system at Yorktown, and for the interval November 18, 1981 to Febru- 
ary 2, 1982. 

Figures 4, 5 and 6, in Section 2.6, show the daily user rating of response 
time of the 3 VM systems at Yorktown, on a monthly average basis, from 
June 82 to June 83 together with the hourly response time, hourly expan- 
sion factor and downs per day. 


2.6. Figures for generated report and graphs 
FIGURE 1 


DATE: 83/07/20 


RESPONSES (INCLUDING NO OPINION): 82 NO OPINION: 17 21% 


VERY VERY NUMB AVG 
SERVICE GOOD GOOD FAIR POOR POOR RESP RATING 
RESP. T. Tie 21 28% 18 35% 23 9% 6 11% «7 65 2.69 
RESP. T. P>50 0% OO 0% O O0% 0 0% O 100% 1 1 5.00 


DATE: 83/07/01 TO 83/07/20 


RESPONSES (INCLUDING NO OPINION): 917 NO OPINION: 226 25% 


VERY VERY NUMB AVG 
SERVICE GOOD GOOD FAIR POOR POOR RESP RATING 
RESP. T. 13% 88 28% 193 31% 214 13% $93 15% 103 691 2.90 


RESP. T. P>50 8% 2 15% 4&4 46% 12 8% % 23% 6 26 3.23 
DATE: 83/06/22 TO 83/07/20 


RESPONSES (INCLUDING NO OPINION): 1392 _ NO OPINION: 347 25% 


VERY VERY NUMB AVG 
SERVICE GOOD GOOD FAIR POOR POOR RESP RATING 
RESP. T. 12% 130 28% 297 29% 305 13% 139 17% 174 1045 2293 


RESP. T. P>50 10% - 3 17% 5 47% 14 7% 2 20% 6 30 3210 


Avg. user rating of resp. time 
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Linear fit coeffictents 


Figure 2 


Prime shift YKTVMV 
Time interval: (81/11/18 to 82/02/05) 


0.5 0.6 0.7 
Daily average response time 


Ordinate to the origin: .70 


Slope: 3.86 


R Squared factor: .671 


Daily avg. resp time for a 2.5 avg. user rating of resp. time: .47 
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Figure 3 


Prime shift YKTVMV 
Time interval: (81/11/18 to 82/02/05) 
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Daily average expansion factor 
Linear fit coefficients 
Ordinate to the origin: .35 
Slope: .67 
R Squared factor: .776 
Daily avg. exp. factor for a 2.5 avg. user rating of resp. time: 3.22 
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3. ON-LINE OPINION SURVEYS OF USER AND EMPLOYEE SATISFAC- 
TION 


This facility has been used to conduct the Annual Survey of Yorktown Com- 
puter Services and the IBM Research Division Opinion Survey. 


3.1 Introduction 


In response to a system command SURVEY the user is presented, in full 
screen mode, with the pertinent survey questions and offered the opportu- 
nity to enter related comments. At the end of the session the user's 
replies are transmitted to a collecting virtual machine for tabulation and 
report generation. 


3.2 Objectives 


The main objectives of on-line surveys are the capability of conducting 
surveys on a very short notice, without all the hassle of related paper- 
work and mailing, and the ability to generate automatic summary reports of 
answers either on a partial progressive or final basis. These objectives 
have been achieved only partially since in most cases not all the survey 
participants have access to a computer terminal, are familiar with the 
pertinent techniques, or are willing to take a survey on line preferring 
instead the conventional way of filling a paper form. Reduction in the use 
of paper, greater flexibility for the user to file his survey, improved 
security, elimination of ballot stuffing and greater appeal to certain 
users are some of the additional advantages of on-line surveys. 


3.3 General description 


The facilities covered in this paper operate under VM/SP and are func- 
tional in tube type terminals only. The different components were written 
in CMS EXEC2 language. Full screen support is provided by 1083270 (The 
Display Input/ Facility). There are three main components to the 
facility, namely: 

A system command, named SURVEY, to be invoked by the user in order to 

take the survey. 

A receiving station or collector where the surveys are transmitted to 

and collected. 

Postprocessing programs for tabulation. 


3.4 Flexibility to the user and human factors 


The questions of the survey may be answered by the user in any order. An 
index permits the selection of any desired question. Any or all the 
answers may be reviewed at any time and/or before transmission. The sur- 
vey may be ended by the user at any time with the option of continuing lat- 
er. This enables him to take the survey in several sessions if he so 
desires or finds it necessary. However this feature is not available in 
surveys about very sensitive or confidential matters for security 
reasons. Comments may be entered by means of the participant's preferred 
CMS editor. 

After the invocation of the command the user is presented with an intro- 
duction that contains instructions to the survey. The survey itself con- 
sists of the displays of a certain number of frames containing the survey 
questions and instructions to the user on how to proceed. Defined PF keys 
permit him to slide from display to display. There is little room for 
operational errors since input fields are the only ones that are unpro- 
tected, furthermore the autoskip feature of full screen mode is in effect 
throughout the survey. 


3.5 System command SURVEY 


SURVEY is the CMS system command that initiates the survey. It is opera- 
tional only during the duration of the survey. The users' replies and 
comments are written in a CMS file in one of his minidisks. At the end of 
the survey the users’ answers are transmitted to a receiving virtual 
machine for collection. The conduction of a new survey requires the 
change of the input data file containing the questions and 1083270 full 
screen control characters. 
In the case of confidential surveys and for security reasons the following 
are additional restrictions: 
The availability of the command SURVEY is restricted to business hours. 
During the users' session his answers are kept in temporary disk space. 
The advantages are that the information is not accessible, and it is 
erased in case of a system crash. The obvious disadvantage is that the 
information is lost if the system crashes with the result of the user 
having wasted his effort. 


3.6 The collector 


The survey files are read as soon as they arrive to the collector virtual 
reader. The originating user id is detected at this time and recorded with 
the aim of precluding ballot stuffing. In the case of confidential sur- 
veys and for security reasons the following additional features are imple- 
mented: 
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The user id undergoes checks for eligibility (i.e if it is a regular IBM 
employee) and for previous participation. Despite these checks the par- 
ticipant remains anonymous. 

Rejected files are discarded and the originator is notified electron- 
ically of the reasons for the rejection. 

Accepted survey files are assigned a sequential number and encrypted pro- 
vided the virtual machine is operating in enciphering mode. Encryption is 
done by means of IPS/CMS Cryptographic Program (IBM IUP 5796-PPK). The 
originator is notified by electronic mail about his survey's acceptance. 
An enciphering key is used for encryption of the files. Enciphering mode 
is established at the time the enciphering key is entered manually by the 
survey administrator. The enciphering key resides in virtual memory only 
for security reasons. The key is tested to determine if it is the right 
key. Wrong keys are rejected. The right key is the key entered by the sur- 
vey administrator at key establishment. Key establishment is done only 
once by means of a key generation program. No record of the key remains in 
disk storage. The collector notifies the survey administrator when is not 
operating in enciphering mode (i.e. every time the system is brought up). 
Multiple checks exclude the possibility of a survey file being lost. In 
this regard the collector notifies the survey administrator and the pro- 
grammer about the nature of any malfunction and then logs itself off. 
Access to the receiving virtual machine is restricted to the survey admin- 
istrator and authorized employees. 


3.7 The postprocessing programs 


This component completes the cycle of automating the conduction of 
surveys. 

There are currently two postprocessing programs. One for confidential 
surveys (i.e. an IBM Opinion SURVEY), and another for surveys where 
confidentiality is not so critical. In either case the programs require 
information about the structure and format of the survey. The final output 
consists of summary reports containing statistical tabulations of the 
survey's answers, plus files where the users’ comments are classified and 
appended. Postprocessing may be done on several sessions as the survey 
progresses or at its conclusion. In the case of the IBM Research Division 
Opinion Survey the postprocessing component is an interface to TOSS (Total 
Opinion Survey System) developed by IBM Corporate. 

The following steps and considerations apply to the case of confidential 
surveys: 

The survey files are deciphered. The cipher key is entered manually by the 
survey administrator and it resides in virtual memory only. Generated 
reports are encrypted and all deciphered files are erased before the ses- 
sion is ended. Access to the postprocessing programs is restricted to the 
survey administrator and authorized employees. 


3.8 Usage 


The survey facility has been used at the Thomas J. Watson Research Center 
of IBM since 1981 to conduct the Annual Survey of Yorktown Computing Ser- 
vices. It was also used to conduct the IBM Research Division Opinion Sur- 
vey, Fall 1982. These surveys were also available in paper form. 

1100 employees participated in the IBM Research Division Opinion Survey. 
Of this 517 took the survey on line (47% of the total). 

356 users took the 1982 Annual Survey of Yorktown Computing Systems. 
Approximately half of them (177) participated on line. 


3.9 Demonstration 


The following example demonstrates through captured spooled console 
images the case when a user invokes the SURVEY command, replies to 
questions 1 and 2, selects from the index question 9, answers question 9, 
enters comments to question 9, answers question 10, ends the survey and 
transmits his answers. 
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The survey introduction is displayed following the invocation of the 
system command SURVEY. In this case the introduction extends over 
several frames. Only the first frame is displayed. 


ANNUAL SURVEY OF YORKTOWN COMPUTING SERVICES 


In order to determine user satisfaction with our services, 
we are conducting this survey. 


Please base your answers only upon your experiences during 
the past three months. 

Do not answer questions which do not apply to you or for 
which you have no opinion. 


This survey consists of 39 questions. You can proceed to the 

following question by pressing the ENTER key or go back to 
the preceding question by pressing PF7. You are not limited 

to this sequential mode but may move to any question you 

wish by pressing PF2 and making a question number selection 
from a question index. You may go back to enter your 
response to a previous question that was left unanswered, or 

to change a previous response. 


Whenever a question applies to more than one service, the 


Press ‘ENTER or PF8' to CONTINUE (Next frame); 
PF3 (Quit); 


The first question is displayed after the introduction. 
The cursor is positioned at the first input field. 


QUESTION 1 OF 39 


Which one of these computers have you used the most during 
the past three months? 


==> 


==> 


==> 


==> 


==> 


==> 


(Enter an x in appropriate field.) 


VM-V 

VM-T 

VM-X 

MVS/TSO 

IBM PC 

IBM PC connected to VM 


PF2 (For question index); Press ‘ENTER‘or PF8 to CONTINUE; 
PF3 (End); PF4 (To enter comments); PF7 (To view Introduction); 


After a reply has been entered in the first field the 
cursor moves to the next input field (autoskip feature). 


QUESTION 1 OF 39 


Which one of these computers have you used the most during 
the past three months? 


(Enter an x in appropriate field.) 


==> x VM-V 
==> VM-T 
==> VM-X 
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==> MVS/TSO 
==> IBM PC 


==> IBM PC connected to VM 


PF2 (For question index); Press ‘ENTER'or PF8 to CONTINUE; 


PF3 (End); PF4 (To enter comments); PF7 (To view Introduction); 


Question 2 is displayed next following the replies to question 1. 


QUESTION 2 OF 39 


Where are you located? 


(Enter an x in appropriate field.) 


==> x Yorktown 
==> Eastview 


==> Another IBM location (specify): 


PF2 (For question index); Press ‘ENTER’ or PF8 to CONTINUE; 
PF3 (End); PF4 (To enter comments); PF7 (Previous question); 
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The survey index is displayed after pressing the PF4 key. 
The index extends over several frames. Only the first frame 
is displayed. Question 9 is then selected. 
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11. 


ENTER QUESTION NUMBER ===> 9 


So 


SURVEY INDEX 


Which one of these computers have you used the most during the past 
three months? 

Where are you located? 

For what purposes do you use Yorktown's VM and/or MVS computing 
systems? . 

For what purposes do you use the IBM PC? . 

To what extent has your PC usage affected your usage of Yorktown's 

VM and MVS computing systems? 

How do you rate average system response for most of the work that you do 
on prime shift? 

How do you rate average system response for your prime shift interactive 
work that requires moderate to large amounts of computer resources (e.g., 
system responses exceeding several seconds)? 

How do you rate turnaround time for your prime shift batch work? 

How do you rate turnaround time for your printouts? 

How do you rate system reliability (number of times the system goes 

down during your working session)? 

How do you rate system availability (fraction of time 

system is available when you need to work)? 


AND PRESS ENTER 
PF3 (Quit Index); PF7 (Previous frame); PF8 (Next frame); 


Next the user enters his replies to question 9 and then decides 
to enter some comments. To do this he presses the PF4 key. 


QUESTION 9 OF 39 


How do you rate turnaround time for your printouts? 


(Enter: 1 for very good, 2 for good, 3 for fair, 4 for poor, 5 for very poor. 


==> 
<=> 
==> 
==> 
==> 
==> 
==> 
==> 


==> 


WN = aN bd 


Leave blank for no opinion.) 


Versatec/Benson (DRAFT) printers 
APS-5 photocomposer 

3800 . 

Self-service printers at Yorktown 
6670 printers at Yorktown 
Confidential output at Yorktown 

At Eastview 

At other IBM locations over network 
IBM PC 


PF2 (For question index); Press ENTER’ or PF8 to CONTINUE; 


PF3 (End); PF4 (To enter comments); PF7 (Previous question); 
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The intention to enter comments was signaled at question 9 by 
pressing the PF4 key. the user enters now his comments, and 
files. He is then returned to the survey questionnaire. 


C$9$ MEMO Al F 80 TRUNC=80 SIZE=3 LINE=3 COL=1 ALT=2 
XEDIT: 


00000 * * * TOP OF FILE * * * 
00001 COMMENTS TO 9: How do you rate turnaround time for your printouts? 
00002 
00003 Printouts are sometimes misplaced by the operator 

ea Sle Ade wG se Ree ee Ob nee ee See ee Be ee eee Oar ee laa 
00004 * * * END OF FILE * * * 


===> file 
XEDIT/ykt 1 FILE 


After answering question 10 the user decides to end the 
survey by pressing the PF3 key. 


QUESTION 10 OF 39 
How do you rate system reliability (number of times the system goes 
down during your working session)? 


(Enter: 1 for very good, 2 for good, 3 for fair, 4 for poor, 5 for very poor. 
Leave blank for no opinion.) 


==> 2 VM-V 

==> 2 VM-T 

==> 2 VM-X 

==> 1MVS/TSO 

==> 2 Network - among Yorktown systems 

==> 2 Network - between Yorktown and other IBM locations 
==> 3 MSS 

==> 1 IBM PC 


PF2 (For question index); Press ‘ENTER’ or PF8 to CONTINUE; 
PF3 (End); PF4 (To enter comments); PF7 (Previous question); 
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The user is presented with a multiple choice menu. 
He then chooses option 1. 


Your replies to this survey are stored only on your disk. 
What would you like to do with them? 


1. Transmit them (for tabulation with other users’ replies) 
and erase them from my disk. 


2. Do not transmit them, but keep these replies on my disk 
so that | may continue the survey at some other time. 


3. Do not transmit my replies and erase them from my disk. 


4. Do not transmit my replies. Let me resume taking the 
survey at the point where | left. 


5. Do not transmit my replies. Let me review the survey 
starting at question 1. 


ENTER SELECTION NUMBER ===> 1 AND PRESS ENTER 


3.91 Format of the survey replies in the Collector. 


The following data, transmitted to the receiving virtual machine for tabu- 
lation, corresponds to the example in the demonstration in section 3.9. 
The data has undergone some preliminary processing and is ready for 


postprocessing. 


YKTVMT TOLABA TUESDAY 07/26/83 13:19:48 13:22:33 
ANSWERS TO QUESTION 1 


Xx 

ANSWERS TO QUESTION 2 

X 

COMMENTS TO QUESTION 3 

COMMENTS TO 3: For what purposes do you use Yorktown's VM and/or MVS 
computing systems? 

ANSWERS TO QUESTION 9 

ee ae ea 

COMMENTS TO QUESTION 9 

COMMENTS TO 9: How do you rate turnaround time for your printouts? 


Printouts are sometimes misplaced by the operator 
ANSWERS TO QUESTION 10 
222 252-30 


